{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9f66a289-a4d3-450a-8d4e-f886b4df19f4",
   "metadata": {},
   "source": [
    "# Llama-index load data\n",
    "https://docs.llamaindex.ai/en/stable/understanding/loading/loading/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6e25e0b-ca80-47fe-bdf1-5f80b55cbd2c",
   "metadata": {},
   "source": [
    "## This ingestion pipeline typically consists of three main stages\n",
    "\n",
    "- Load the data\n",
    "- Transform the data\n",
    "- Index and store the data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76385c13-8fb2-46dd-a637-8ce6786caad7",
   "metadata": {},
   "source": [
    "### Loading using SimpleDirectoryReader\n",
    "The easiest reader to use is our SimpleDirectoryReader, which creates documents out of every file in a given directory. It is built in to LlamaIndex and can read a variety of formats including Markdown, PDFs, Word documents, PowerPoint decks, images, audio and video."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "3eb0a87f-03f6-4c74-9db5-12071da5eca1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting docx2txt\n",
      "  Using cached docx2txt-0.8.tar.gz (2.8 kB)\n",
      "  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hBuilding wheels for collected packages: docx2txt\n",
      "  Building wheel for docx2txt (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for docx2txt: filename=docx2txt-0.8-py3-none-any.whl size=3959 sha256=16907b5dee9a0c68c38e190ad9243c04aed3506b026c6799ef0bf5a4748126bf\n",
      "  Stored in directory: /Users/laobao/Library/Caches/pip/wheels/22/58/cf/093d0a6c3ecfdfc5f6ddd5524043b88e59a9a199cb02352966\n",
      "Successfully built docx2txt\n",
      "Installing collected packages: docx2txt\n",
      "Successfully installed docx2txt-0.8\n"
     ]
    }
   ],
   "source": [
    "!pip install docx2txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "150c0e8a-f6b6-46e7-a6f8-e75380b94151",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core import SimpleDirectoryReader\n",
    "\n",
    "# 如果目录中有docx文件，需要安装docx2txt\n",
    "documents = SimpleDirectoryReader(\"../datasets\").load_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a0d188e-c485-4e26-82ee-3f65969b2b3e",
   "metadata": {},
   "source": [
    "## Setup LLMs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "57eeded5-5b9b-4f87-8896-8ddbe52b9975",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " The term \"digital thread\" refers to a concept in the field of product lifecycle management (PLM) and systems engineering. It represents an integrated, information-driven approach to the design, manufacturing, operation, and maintenance of complex products and systems. The digital thread connects all the digital models and simulations of a product throughout its lifecycle, providing a seamless flow of information and insights across different stages and stakeholders.\n",
      "\n",
      "Key aspects of the digital thread include:\n",
      "\n",
      "1. **Integration**: It integrates data from various sources, such as CAD (Computer-Aided Design) models, PLM systems, ERP (Enterprise Resource Planning) systems, and IoT (Internet of Things) devices, to create a unified view of the product.\n",
      "\n",
      "2. **Traceability**: The digital thread allows for traceability, meaning that it enables stakeholders to track the lineage of a product's design, changes, and decisions from conception to disposal.\n",
      "\n",
      "3. **Real-time Data**: It provides real-time data and insights, which can be used to make informed decisions, optimize processes, and improve product quality.\n",
      "\n",
      "4. **Collaboration**: The digital thread facilitates collaboration among different teams and departments, as everyone has access to the same up-to-date information.\n",
      "\n",
      "5. **Lifecycle Management**: It supports lifecycle management by ensuring that the product's digital information is consistent and up-to-date at every stage, from design and engineering to manufacturing, operations, and maintenance.\n",
      "\n",
      "The digital thread is a critical component of digital transformation initiatives, as it enables companies to leverage digital technologies to improve efficiency, reduce costs, and enhance the overall quality of their products and services. It is particularly important in industries with complex products, such as aerospace, automotive, and heavy machinery, where the ability to manage and analyze vast amounts of data is essential for success.\n"
     ]
    }
   ],
   "source": [
    "from llama_index.llms.openai import OpenAI\n",
    "from llama_index.core.llms import ChatMessage\n",
    "from llama_index.llms.openai_like import OpenAILike\n",
    "\n",
    "llm = OpenAILike(api_key=\"sk-1234\", api_base=\"http://1.15.125.13:3033/v1\",model=\"deepseek-chat\")\n",
    "\n",
    "\n",
    "response = llm.complete(\"What is digital thread?\")\n",
    "print(str(response))\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3cd677a8-9058-4190-8a9a-68ba79a021c4",
   "metadata": {},
   "source": [
    "## stream out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "9ce3d02c-5221-40cc-ace6-1648420eac8a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# gen = llm.stream_complete(\"what is digital engineering?\")\n",
    "# for response in gen:\n",
    "#     print(response.text, end=\"\", flush=True)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94133a91-f184-4439-bdc7-e8c813f64297",
   "metadata": {},
   "source": [
    "### Transformations\n",
    "After the data is loaded, you then need to process and transform your data before putting it into a storage system. These transformations include chunking, extracting metadata, and embedding each chunk. This is necessary to make sure that the data can be retrieved, and used optimally by the LLM."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "134b1c99-be3c-43a2-8c66-7a61d4607887",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/laobao/opt/anaconda3/envs/llama-index/lib/python3.10/site-packages/transformers/utils/generic.py:309: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.\n",
      "  _torch_pytree._register_pytree_node(\n"
     ]
    }
   ],
   "source": [
    "from llama_index.core.node_parser import SentenceSplitter\n",
    "from llama_index.core import Settings\n",
    "from llama_index.embeddings.huggingface import HuggingFaceEmbedding\n",
    "\n",
    "embed_model = HuggingFaceEmbedding(model_name=\"BAAI/bge-small-en-v1.5\")\n",
    "\n",
    "Settings.llm = llm\n",
    "Settings.embed_model = embed_model\n",
    "\n",
    "text_splitter = SentenceSplitter(chunk_size=512, chunk_overlap=10)\n",
    "\n",
    "from llama_index.core import Settings\n",
    "Settings.text_splitter = text_splitter\n",
    "\n",
    "index = VectorStoreIndex.from_documents(\n",
    "    documents, transformations=[text_splitter]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52181033-0dc4-431d-a1d2-74c85adbe738",
   "metadata": {},
   "source": [
    "> 使用OpenAILike， 在上下文使用llm对象，需要使用Settings对象\n",
    "```\n",
    "Settings.llm = llm\n",
    "Settings.embed_model = embed_model\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "18e2ceb6-84f0-4d8d-9d29-03a65dbb6c38",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
